logo



Approach


This analysis estimates the impact of Balto usage on Outbound transfer rate for SelectQuote agents. To estimate this impact, the following process is used:

  1. Estimate a statistical (mixed logistic) model that is a function of who the agent is, whether the agent is a Balto user, time, their skill level, and vendor campaign. We additionally interact the indicator for being a Balto user with agent skill level, allowing the effect of Balto to vary across skill level.
  2. Calculate a 95% confidence interval around the estimated Balto effect for each skill level, then translate the estimates to a percentage deviation from the pre-test average for each respective skill.
  3. Visualize the estimated effects.

We employ this approach to account for all potential confounding factors linked to the KPI of interest that could bias results in the presence of insufficient group randomization. Taking this approach also avoids unnecessary aggregations and maximizes available information to make statistical inference.

We find a positive statistically significant effect for Dual Skilled agents, and a statistically insignificant negative effect for Specialty agents.

Results



Summary Tables


Dual Skilled Agents

Specialty Agents


Visualizations


Technical Details


Model Output

  Transfer Rate
Predictors Odds Ratios 95% CI
Intercept 0.01 *** 0.00 – 0.01
Balto User = Yes 2.57 * 0.94 – 6.99
Agent Level = Specialty 3.35 ** 1.05 – 10.75
Balto User = Yes * Agent
Level = Specialty
0.39 0.11 – 1.34
Linear Time Trend 0.76 *** 0.64 – 0.91
Random Effects
σ2 3.29
τ00 Agent 0.00
τ00 VendorCampaignName 0.00
N VendorCampaignName 18
N Agent 28
AIC 1485.773
log-Likelihood -735.887
  • p<0.1   ** p<0.05   *** p<0.01

Marginal Effects

Data


 

Balto.ai